System and method for inputting text into electronic devices

ABSTRACT

A text prediction engine, a system comprising a text prediction engine, and a method for generating sequence predictions. The text prediction engine, system and method generate a final set of sequence predictions, each with an associated probability value.

The present invention generally relates to a text prediction engine,system and method for inputting text into electronic devices.

Many existing inventions provide methods for enhancing text entry forusers of electronic devices, utilising a wide range of techniques.However, it is believed that the disclosed system is the first toapproach the problem of predicting the intended written text of a userusing a robust, fully integrated probabilistic model.

In a first aspect of the present invention, there is provided a textprediction engine comprising: at least one model configured to generatefrom an evidence source a first set of sequences with associatedprobability estimates; a probability generator configured to receive theset of first sequences with associated probability estimates and togenerate a set of sequence predictions with associated probabilityvalues, wherein the probability values are approximately normalised overall possible sequence predictions generated by the probability generatorgiven all of the possible sequences received by the probabilitygenerator.

Preferably, the text prediction engine comprises a prior modelconfigured to generate a second set of sequences with associatedprobability estimates.

Preferably, the at least one model is configured to generate a first setof sequences based on the evidence source and an uncertainty in theevidence source. Preferably, the probability generator is configured toreceive the first and second sets of sequences with associatedprobability estimates.

The probability generator is preferably configured to estimate anormalisation factor for the probability values by summing theprobability values for the n most probable sequence predictions and aconstant representative of the probability values for the remainingpossible sequence predictions. The constant represents the probabilityvalues for the remaining possible sequence predictions generated by theat least one model and by the prior model.

The at least one model preferably comprises a plurality of modelsconfigured to generate a plurality of first sets of sequences withassociated probability estimates. In one embodiment, the plurality ofmodels are configured to generate the plurality of first sets ofsequences from a plurality of evidence sources.

Preferably, the text prediction engine is part of a system and the userinputted text can be input into the system by one or more of userselection, character entry or voice recognition.

The text prediction engine is configured to weight the probabilityvalues of the sequence predictions according to the probability that thecorresponding model comprises a given context sequence. In oneembodiment, the plurality of models comprise a plurality of languagemodels corresponding to a plurality of different languages and theprediction engine is configured to weight highest the probability valuesof the sequence predictions corresponding to the language model relatingto the most likely language of user inputted text.

Each evidence source may be modelled by a corresponding model togenerate the set of sequences with associated probability estimates. Theprobability generator is preferably configured to treat each evidencesource as conditionally independent of all other evidence sources, giventhe set of sequence predictions.

In a preferred embodiment of the text prediction engine, the at leastone model comprises a context model and an input model, each of which isconfigured to receive text input by a user and to generate a set ofsequences and associated probability estimates; and the prior modelcomprises a target prior model configured to generate a set of sequencesand associated probability estimates. The input model preferablycomprises a candidate model and a language model. The context modelpreferably comprises a candidate model and a prefix match model. Thetarget prior model preferably comprises a character model and a unigrammodel.

In a second aspect of the present invention, there is provided a systemcomprising: a user interface configured to receive text input by a user;a text prediction engine configured to receive the text input from theuser interface and to generate a set of sequence predictions withassociated probability values, wherein the probability values areapproximately normalised over all possible sequence predictions; whereinthe text prediction engine is further configured to provide the set ofsequence predictions to the user interface.

Preferably, the input model comprises a candidate model and a languagemodel. Preferably, the context model comprises a candidate model and aprefix match model. Preferably, the target prior model comprises acharacter model and a unigram model.

In a third aspect of the invention, there is provided a method forprocessing user text input comprising: receiving text input into a userinterface; generating, using a text prediction engine, a set of sequencepredictions and associated probability values, wherein the probabilityvalues are approximately normalised over all possible sequencepredictions; providing the set of sequence predictions to the userinterface.

Generating normalised probability values preferably comprises estimatinga normalisation factor for the probability values by summing theprobability values for the n most probable sequence predictions and aconstant representative of the remaining possible sequence predictions.

The method may further comprise displaying the set of sequencepredictions on the user interface for user selection. Preferably, thesequence predictions are ordered by the text prediction engine forordered display by the user interface. The sequence predictions may beprovided to the user interface only if their corresponding probabilityvalues are each greater than or equal a first threshold value.Similarly, the system may be configured to provide to the user interfacepredictions only if their corresponding probability values are eachgreater than or equal a first threshold value.

Preferably, the at least one of the set of sequence predictionscorresponds to an adjusted or corrected version of text input by a userinto the user interface.

The method may further comprise inputting automatically a sequenceprediction which has a probability value greater than or above a secondthreshold value. Similarly, in one embodiment, the system may beconfigured to input automatically a sequence prediction which has aprobability value greater than or above a second threshold value.

The probability generator used in the method preferably comprises aplurality of models, each configured to generate a set of sequencepredictions and associated probability values, and the probabilityvalues are weighted according to the probability that the correspondingmodel comprises a given context sequence.

The present invention also provides a computer program productcomprising a computer readable medium having stored thereon computerprogram means for causing a processor to carry out the methods asdescribed above.

The invention also relates to a text prediction engine for generatingsequence predictions, and a system and method for generating sequencepredictions for display and user selection. In one embodiment, theinvention relates to a system for automatically correcting anerroneously entered sequence, and a method for doing so. In a preferredembodiment, the present invention provides a text prediction engine, asystem and method for generating a final set of sequence predictions,each with an associated probability value, by combining any number ofseparate probability estimates for sequence intentions. The presentprediction engine, system and method can therefore provide predictionswhich are based on any number of independent evidence sources. This canbe achieved by the assignment of a true probability to each of thepredicted sequences, rather than just a ranking over sequences. Byassigning true probability values, evolution of the probabilitiesassigned to different terms can be analysed and probabilities for agiven term, or set of terms, at two different points in time can becompared. This means that a-priori thresholding can be utilised tomoderate the behaviour of the system given its “confidence” in aparticular prediction. For instance, a predicted sequence might only bedisplayed, or an automatic correction made, if the system estimates theprobability of accuracy at over 0.75, or in other words if there is atleast a 75% chance that the predicted sequence is accurate. This type ofinference is impossible if some kind of ad-hoc score is used to rankelements, as such values cannot be reliably compared between sequencesover time.

To generate the true probability values, the present inventionpreferably provides a means for efficiently approximating thenormalisation sum over all sequences.

The present invention will now be described in detail with reference tothe accompanying drawings, in which:

FIG. 1 is a schematic of a high level prediction architecture accordingto the invention;

FIG. 2 is a schematic of an example of a preferred predictionarchitecture according to the present invention.

DEFINITIONS

-   -   character—a symbol representing an atomic orthographic unit    -   character set—a finite collection of characters    -   sequence—a finite length, ordered string of characters    -   prefix—one sequence s is a prefix of another s′ if, beginning        with the first character in each sequence, there is a        contiguous, one-to-one mapping and length(s)≦length(s′).    -   proper prefix—one sequence s is a proper prefix of another s′        ifs is a prefix of s′ and length(s)<length(s′).    -   language—a (usually infinite) set of sequences characteristic of        a particular written or spoken community    -   text—written data drawn from one or more languages    -   system—the subject of this disclosure    -   user—an intentional agent interacting with the system

In general, but not exclusive terms, the system of the invention can beimplemented as shown in FIG. 1. FIG. 1 is a block diagram of one highlevel text prediction architecture according to the invention. Thesystem comprises a text prediction engine which generates a set of mostlikely sequence predictions S_(F) intended to be entered by the user.Each sequence prediction has a probability value associated with it.

As can be seen from FIG. 1, the text prediction engine preferablycomprises a plurality of trained models M₁, M₂, M₃ etc. that will beused to make probabilistic inferences from a plurality of evidencesources e₁, e₂, e₃, etc.; and a probability generator (PG). However, inother embodiments, there may be a single trained model and singleevidence source.

There are arbitrarily many potential sources of evidence e₁, e₂, etc.Some examples of which include:

-   -   the sequence the user has already entered    -   the term/phrase the user is currently entering    -   stored, historical sequences entered by the user    -   the user's native language    -   the specific style of language being entered    -   the application in which the current sequence is being entered    -   in a messaging environment, the intended message recipient    -   the time/date    -   the location of the device hosting the system

General Model

The aim of the system is to rank sequences in a given subset of alanguage by the likelihood that the user intended/intends to enter thatsequence. In probabilistic terms, this equates to a ranking oversequences in a set S governed by the following:

P(s∈S|e,M)  (1)

where e is the observed evidence, and M is the set of trained modelsthat will be used to make probabilistic inferences. In other words, thesystem will estimate the conditional probability, over the set of allsequences from which predictions can be drawn, of the sequence given theevidence e. The target sequence is denoted by s.

In order to simplify the process of combining predictions from differentdata sources, in a preferred embodiment, the target sequence, s, isdefined as a prediction which comes from a specific data source.

Each model in M is trained on a particular data source. Thus, aparticular data source is represented by a model in M, and the set S inexpression (1) ranges over all distinct terms (or sequences) generatedby the models in M. A model is queried to provide a predicted term. Thisterm is associated with the model from which it was drawn, and istherefore different from a lexically identical term drawn from adifferent model, due to its association with the model from which it wasdrawn. This association can be implicit in the data. However, the termcould be tagged with an identifier associated with the model from whichit has been drawn.

In this preferred process of combining predictions, two otherwiseidentical predictions that come from different data sources areconsidered different. To combine sequences from different models to geta list of predictions, the sequences are simply ranked with duplicatepredictions removed. In the preferred implementation, the most probableestimate is retained for a given lexical term/sequence and any (lessprobable) lexical duplicates are discarded.

By way of a non-limiting example, if M comprises two context languagemodels, French (LM_(French)) and English (LM_(English)), the term “pain”will probably occur in both, and will appear twice in S, once linkedwith the French model and once with the English model. This will resultin two separate estimates for the term “pain” given a particular set ofevidence (where the evidence in this case is the context which precedesthe predicted term “pain”).

These estimates refer to two distinct sequences (one from French, onefrom English); however, because they are lexically identical, it is notnecessary to present them both to the user. Thus, in accordance with thepreferred embodiment, the most probable estimate for a given lexicalsequence is retained and any lexical duplicates are discarded.

In order to rank sequences in a given subset of a language by thelikelihood that the user intended/intends to enter that sequence, theconditional probability of expression (1), P(s∈S|e, M), needs to becalculated. To determine this probability, the expression is rearrangedusing Bayes' rule to yield:

$\begin{matrix}\frac{{P\left( {\left. e \middle| s \right.,M} \right)}{P\left( s \middle| M \right)}}{P\left( e \middle| M \right)} & (2)\end{matrix}$

and marginalised over target sequences in the denominator to yield:

$\begin{matrix}\frac{{P\left( {\left. e \middle| s \right.,M} \right)}{P\left( s \middle| M \right)}}{\sum\limits_{j = 1}^{S}{{P\left( {\left. e \middle| s_{j} \right.,M} \right)}{P\left( s_{j} \middle| M \right)}}} & (3)\end{matrix}$

In order to compute P(e|s, M), in a preferred embodiment, it is assumedthat the evidence can be separated into non-overlapping sets, [e₁ . . .e_(N)], that are independently generated from some distribution under anassociated model [M₁ . . . M_(N)], given the target sequence. Thisindependence assumption can be written as:

P(e|s,M)=Π_(i=1) ^(N) [P(e _(i) |s,M _(i) ∈M)]  (4)

and stated as:

-   -   Assumption 1: Evidence can be separated into distinct sets, such        that the evidence in each set is conditionally independent of        all others, given the target sequence;        where each e_(i) has a model M_(i) associated with it. This will        allow a framework to be constructed in which it is possible to        combine arbitrarily many sources of evidence in a        computationally efficient manner. In a preferred embodiment, a        model R∈M is associated with a target sequence prior. Given this        assumption we can restate (3) as follows:

$\begin{matrix}\frac{{P\left( s \middle| R \right)}{\prod\limits_{i = 1}^{N}{P\left( {\left. e_{i} \middle| s \right.,M_{i}} \right)}}}{\sum\limits_{j = 1}^{S}{{P\left( s_{j} \middle| R \right)}{\prod\limits_{i = 1}^{N}{P\left( {\left. e_{i} \middle| s_{j} \right.,M_{i}} \right)}}}} & (5)\end{matrix}$

Thus, in a preferred embodiment, the conditional probability ofexpression (1) can be computed by computing the target sequence prior,P(s|R), and each of the evidence likelihoods, P(e_(i)|s, M_(i)).

The denominator in expression (5) is constant with respect to s andtherefore does not affect the ranking, rather it is a normalisationfactor on computed probability values. In a preferred embodiment thisvalue is estimated as the sum of a subset of the most probable sequencesand a constant, to overcome the problem of having to compute theconditional probability over all sequences in S (see expressions 13-15below). This approach is reasonable due to the Zipfian nature of manynatural language phenomena, where a minority of likely events carry themajority of the probability mass. The Zipfian distribution is aninstance of a power law distribution, in which the frequency of a givenevent is approximately inversely proportional to its rank.

Expression (5) provides a principled approach to combining differentsources of evidence about text entry intentions, and in the preferredsystem of the present invention is implemented by a set of trainedmodels R, M₁, M₂, . . . generating a set of sequences S_(R), S₁, S₂, . .. and an associated set of conditional likelihood values R_(R), P₁, P₂,. . . given a source of evidence e₁, e₂, . . . . The model R is used tocompute the prior target sequence probability P(s|R), while each modelM₁, M₂, . . . computes the respective evidence likelihood P(e_(i)|s,M_(i)). Each model outputs a set of sequences S, and a set of associatedconditional probabilities P_(i). Each model M₁, M₂, . . . may compriseone or more sub-models. The probability generator PG takes the sequencesand associated conditional probabilities as input and outputs a finalset of sequences S_(F) associated with probability values P_(F). Theprobability generator PG can combine the predictions as described in thepreferred process above, i.e. the predictions are ranked in order ofprobability and any duplicate predictions are simply removed. The set ofsequences S_(F) associated with the final probability values P_(F) canbe presented, for example in a list format, on a user interface of thesystem, for user review and selection. The user interacts with thesystem by making prediction selections or manipulating the device onwhich the system resides in other ways, thus updating the evidence. Eachmodel R, M₁ . . . M_(N) may be updated as text is input into the system.

The present invention provides two preferred methods for computingevidence likelihoods within a probabilistic framework by marginalisingover candidate interpretations of the evidence, represented in a graphframework, although other methods may be utilised. These two preferredmethods will now be discussed.

Candidate Model 1

When forming an estimate of the likelihood of the evidence from a singleevidence source, P(e_(i)|s, M_(i)), it is often helpful to express themodel in terms of ‘candidates’, which are intermediate stages betweenthe ‘user-intended’ sequence, and the observed evidence. If expressed interms of candidates, the likelihood P(e_(i)|s, M_(i)) can be re-writtenas:

P(e|s,M)=Σ_(j=1) ^(K) P(e|c _(j) ,s,M _(candidate))P(c _(j) |s,M_(sequence))  (6)

where c_(j) is a single candidate, and there are now two submodels of Mfor a given evidence source: the candidate model M_(candidate) and thesequence model M_(sequence). The key assumption here is as follows:

-   -   Assumption 2: The likelihood, under the model in question, can        be expressed as a marginalization over candidates, where the        evidence is conditionally independent of the target sequence        given the candidate.

Applying this assumption, the dependence on s can be dropped from theevidence term:

P(e|s,M)=Σ_(j=1) ^(K) P(e|c _(j) ,s,M _(candidate))P(c _(j) |s,M_(sequence))  (7)

The properties of the candidate model can also be encoded in the form ofgraphical models describing the relationship between the variables andmodels, for example as shown below:

Candidate Model 2

Another variant of the candidate model first transforms the evidencelikelihood using Bayes' rule:

$\begin{matrix}{{P\left( {\left. e \middle| S \right.,M} \right)} = \frac{{P\left( {\left. s \middle| e \right.,M} \right)}{P\left( e \middle| M \right)}}{P\left( s \middle| M \right)}} & (8)\end{matrix}$

In an embodiment, the evidence conditional sequence probability can bere-written as:

P(s|e,M)=Σ_(j=1) ^(K) P(s|c _(j) ,e,M _(sequence))P(c _(j) |e,M_(candidate))  (9)

where c_(j) is a single candidate, and as before, there are twosubmodels of M for a given evidence source: the candidate modelM_(candidate) and the sequence model M_(sequence). In this case, the keyassumption is:

-   -   Assumption 3: The likelihood, under the model in question, can        be expressed as a marginalization over candidates, where the        target sequence is conditionally independent of the evidence,        given the candidate.

Applying this assumption, the dependence on s can be dropped from theevidence term:

P(s|e,M)=Σ_(j=1) ^(K) P(s|c _(j) ,e,M _(sequence))P(c _(j) |e,M_(candidate))  (10)

The graphical model for this version of the candidate model is asfollows:

and the full evidence likelihood is:

$\begin{matrix}{{P\left( {\left. e \middle| s \right.,M} \right)} = \frac{\sum\limits_{j = 1}^{K}{{P\left( {\left. s \middle| c_{j} \right.,M_{sequence}} \right)}{P\left( {\left. c_{j} \middle| e \right.,M_{candidate}} \right)}{P\left( e \middle| M \right)}}}{P\left( s \middle| M \right)}} & (11)\end{matrix}$

Specific Model

Using the general and candidate models, we now present a specificexample of the system with reference to FIG. 2, which illustrates apreferred instantiation of the system in which the prediction enginetakes evidence from two distinct sources: context and input. However, asstated above, the system is not limited to taking the context and inputsources as evidence. If other, or additional, evidence sources are usedthe system will be correspondingly configured to generate predictions onthe basis of such evidence sources.

Informally, the context represents the observed evidence about what theuser has already entered, while the input represents the observedevidence about what the user is currently entering. For instance, inEnglish, if the user has entered the sequence “My name is B”, we mightconsider the context evidence to be the sequence “My name is”, and theinput evidence to be the sequence “B”. This is however, only by way ofexample, and it should be noted that in the most general form, the modelsays nothing concrete about the specific form of the observed evidence.For instance, the input evidence might actually be a series of touchcoordinates from a virtual ‘keyboard’.

As shown in FIG. 2, the evidence (input and context) is used as input tothe prediction engine, within which there are preferably three models R,M_(context), M_(input), each of which is preferably comprised of atleast two sub-models (character model, unigram model; candidate model,language model; candidate model, prefix match model). As can be seen inFIG. 2, the prediction engine preferably comprises a target sequenceprior model R. Although preferred, the system is not limited toembodiments which include a target sequence prior model R.

The target sequence prior model R consists of

-   -   Character Model—implements a distribution over sequences in a        language without the concept of a fixed vocabularly. Usually        implemented as a Markov model over character sequences.    -   A character model is a sequence model built from characters        instead of words. For instance, if the training set is        “explaining”, a unigram character model might look like this:    -   P(e)=0.1    -   P(x)=0.1    -   P(p)=0.1    -   P(l)=0.1    -   P(a)=0.1    -   P(i)=0.2    -   P(n)=0.2    -   P(g)=0.1    -   A trigram character model might be as follows:    -   P(e)=0.1    -   P(x|e)=1.0    -   P(p|ex)=1.0    -   P(l|xp)=1.0    -   P(a|pl)=1.0    -   P(i|la)=1.0    -   P(n|ai)=1.0    -   P(i|in)=1.0    -   P(n|ni)=1.0    -   P(g|in)=1.0    -   Unigram Model—implements a distribution over sequences in a        language without taking context into account, internally        treating each sequence as an atomic entity.    -   For example, if the training set is “the dog chased the cat”,        the corresponding unigram language model might be:    -   P(the)->0.4    -   P(dog)->0.2    -   P(chased)->0.2    -   P(cat)->0.2

The context evidence model M_(context) is comprised of

-   -   Candidate Model—implements a conditional distribution over        context observations given a particular candidate        interpretation.    -   Sequence Model—implements a conditional distribution over        sequences in a language, or set of languages, given a particular        context. In FIG. 2, the sequence model is illustrated as a        language model which, in a preferred embodiment, comprises a set        of language models corresponding to different languages, e.g.        LM_(French), LM_(German), LM_(English), etc.

The input evidence model M_(input) is comprised of

-   -   Candidate Model—implements a conditional distribution over input        observations given a particular candidate interpretation.    -   Sequence Model—implements a conditional distribution over        candidates given an intended target sequence. This model is        illustrated in FIG. 2 as a “prefix match model”.

Each model, including the target sequence prior model R, may be updatedwith user entered text, depending on the scenario. By using dynamiclanguage models, the system is able to more accurately predict a givenuser's intended text sequence.

Each model outputs a set of sequences S_(R), S_(context), S_(input) andassociated probability estimates P_(R), P_(context), P_(input) which areused as input for the probability generator PG. The probabilitygenerator PG combines the probability estimates P_(R), P_(context),P_(input) output from the models to generate a set of probability valuesP_(F) for the final sequence predications S_(F).

The final predictions S_(F) can be displayed to the user via a userinterface for user review and selection, or used by the system toautomatically correct erroneously entered text. Once a prediction hasbeen selected, either automatically or by the user, this input ispreferably added to the context evidence which is used to generatefurther predictions. If instead the user adds further input about thecurrent word, by entering further characters, this is preferably addedto the input evidence to alter the current probabilities assigned to thepredictions.

The particulars of how the specific system of this embodiment may begenerated from the mathematical basis will now be explained.

Instantiating expression (5) with the two evidence sources yields:

$\begin{matrix}\frac{{P\left( s \middle| R \right)}{P\left( {\left. {context} \middle| s \right.,M_{context}} \right)}{P\left( {\left. {input} \middle| s \right.,M_{input}} \right)}}{Z} & (12)\end{matrix}$

where Z=the normalisation constant, approximate to:

Σ_(j=1) ^(|S|) P(s _(j) |R)P(context|s _(j) ,M _(context))P(input|s _(j),M _(input))  (13)

This approximation is implemented in the system as follows. Let usconsider a function z over a set of sequences T, such that:

z(T)=Σ_(j=1) ^(|T|) P(s _(j) |R)P(context|s _(j) ,M _(context))P(input|s_(j) ,M _(input))  (14)

Z is computed as:

Z=z(T)+z({u})*k  (15)

where u represents an “unknown” sequence, and k is an estimate of|S|−|T|, where |S| is the number of sequences in the set of all possibletarget sequences, and |T| is the number of sequences for which at leastone of the underlying evidence models has a “known” estimate. Eachindividual evidence conditional model M will return an estimate ofP(e|u, M), i.e. a distribution over evidence observations given the“unknown” sequence. In essence, this means that each evidenceconditional model is responsible for its own distributional smoothing,but this must be related to k which is proportional to the overallestimated number of “unknown” sequences. In practice, each model will“know” about a set of sequences S′ where S′⊂S, and estimates of P(e|s,M) will be constant and equivalent to P(e|u, M) for all s ∉S′. Smoothingof this nature is the means by which the system takes into account thevarying levels of confidence in the models associated with each evidencesource.

According to expressions (12) and (14), in order to determine theconditional probability P(s∈S|e, M) for the example instantiation of thespecific system, the following estimates are computed: a target sequenceprior P(s|R); a context likelihood P(context|s, M_(context)); and aninput likelihood P(input|s, M_(input)), Each of these estimates and howit may be computed is discussed below.

Target Sequence Prior

The target sequence prior is preferably computed as follows:

${P\left( s \middle| R \right)} = \left\{ \begin{matrix}{P\left( s \middle| R_{unigram} \right)} & {{if}\mspace{14mu} \left( {s \in V} \right)} \\{P\left( s \middle| R_{character} \right)} & {otherwise}\end{matrix} \right.$

where V is the set of sequences contained in R_(unigram) and theimplementation of the models is according to known techniques forconstructing smoothed frequency-based unigram language models andsmoothed Markov chain character models. A number of applicabletechniques for implementing these models are listed below. However,other suitable techniques not listed also apply.

-   -   Smoothed n-gram term or character models (known in the art).    -   Adaptive multi language models as described in <ref: UK patent        application no. 0917753.6>.    -   PPM (prediction by partial matching) language models, as        described in e.g. <ref:Scheffler 2008>.    -   Morphological analysis engines configured to generate sequences        probabilistically from constituent lexical components.

By including a target sequence prior model R, the system provides animproved accuracy of intended sequence prediction. Furthermore, thetarget sequence prior model R enables character-based inference ofunseen target sequences, i.e. the system is better able to infer unknowntarget sequences to approximate across all possible target sequences.

Context Likelihood

The context likelihood, P (context|s, M_(context)), is preferablyestimated by means of the second candidate model to provide expression(16) below. Although this is the preferred means to estimate thelikelihood, the invention is not limited to the likelihood beingestimated in this way.

$\begin{matrix}{{P\left( {\left. {context} \middle| s \right.,M_{context}} \right)} = \frac{\begin{matrix}{\sum\limits_{j = 1}^{K}{P\left( {\left. s \middle| c_{j} \right.,M_{{context} - {sequence}}} \right)}} \\{{P\left( {\left. c_{j} \middle| {context} \right.,M_{{context} - {candidate}}} \right)}{P\left( {context} \middle| M_{context} \right)}}\end{matrix}}{P\left( s \middle| M_{context} \right)}} & (16)\end{matrix}$

Thus, to determine the context likelihood, each of the following arecomputed: a context sequence estimate P(s|c_(j), M_(context-sequence));a context candidate estimate P(c_(j)|context, M_(context-candidate)); acontext prior estimate P(context|M_(context)); and a target sequenceprior estimate P (s|M_(context)). Each of these estimates and how it maybe computed is discussed below.

Context Sequence Estimate

The context sequence estimate P(s|c_(j), M_(context-sequence)) is theprobability of the target sequence s given a particular candidatesequence c_(j), under the context sequence model. Formally the contextsequence model is a function that returns the probability of a targetsequence given a context sequence, i.e. ƒ_(s)(t_(target),t_(context))=P(t_(target)|t_(context), θ_(S)) where θ_(S) are theparameters of the model. Thus, the context sequence probability iscomputed as: P(s|c_(i), S)=ƒ_(S)(s, c_(i)). A wide range of differenttechniques can be used to compute this estimate, for example, smoothedfrequency analysis on the context training data, in a similar way toequation (21), and as described in relation to the target sequence priorestimate. Alternatively, any of the following can be used, individuallyor in combination:

-   -   An n-gram language model (known in the art).    -   An adaptive multi language model as described in <ref: GB patent        application no. 0917753.6>.    -   A PPM (prediction by partial matching) language model, as        described in <ref:Scheffler 2008>.    -   A generative HMM (Hidden Markov Model) probabilistic        part-of-speech tagger <ref:Alias-i. 2008. LingPipe 4.1.0.        http://alias-i.com/lingpipe (accessed Sep. 26, 2011) or        Thede, S. M., Harper, M. P., 1999>    -   A natural language parser configured to return probabilities for        partial sentences, such as RASP <ref: Briscoe, E., J. Carroll        and R. Watson 2006>.    -   A neural network configured to accept as input features        representing the contextual and target sequence, and output        probabilities (known in the art).

The system is not limited to the above techniques; any other techniquewhich can be used to compute the context sequence probability isapplicable.

As stated previously, M_(context-sequence) may comprise a plurality oflanguage models corresponding to a plurality of different languages. Todetermine the conditional probability of equation (16), the conditionalprobability is determined using the language model associated with theterm. As an explanation, the previous example of the prediction term“pain” being drawn from an English language model (LM_(English)) and aFrench language model (LM_(French)) is referred to. In this situation,equation (16) is determined as P (context|pain, LM_(English)) andP(context|pain, LM_(French)), where “Pain” drawn from LM_(French) isdifferent from that drawn from LM_(English), even though the predictionsare lexically identical. By associating the term with the model fromwhich it is drawn, the system simplifies the way in which lexicallyidentical terms are dealt with, since only the most probable term isretained from two or more lexically identical terms. Furthermore, thesystem provides a simplification of the calculation of the conditionalprobabilities of equation (16). This is possible because, althoughlexically identical, the terms may have a different meaning in differentlanguages and can therefore be treated as distinct.

Thus, turning to FIG. 2, the set of terms S_(context) generated by themodel M_(context) may comprise terms from any one of the language models(or candidate models) contained within M_(context).

Context Candidate Estimate

The context candidate estimate P(c_(j)|context, M_(context-candidate))is a function of the form:ƒ_(context-candidate)(t)=P(t|θ_(context-candidate)) where t is anarbitrary sequence and θ_(context-candidate) are the parameters of themodel. Thus, the context candidate conditional estimate is computed as:P(c_(j)|context, M_(context-candidate))=ƒ_(context-candidate)(c_(j)).

In a preferred system, a context candidate is a sequence, and the set ofcontext candidates is represented as a directed acyclic graph (DAG),where each node contains a subsequence consisting of one or morecharacters. Each of the edges is assigned a probability and, in apreferred embodiment, the DAG preferably also has the special propertythat each path is constrained to be the same length. This type ofvariant DAG is termed in this document as a probabilistic, constrainedsequence graph (PCSG). Each individual candidate sequence is thenrepresented by a unique path through the PCSG and the context candidatemodel function return value for a given candidate is computed as theprobability of its representative path.

Formally, a PCSG consists of a 4-tuple containing a set of nodes N, aroot node r, a set of directed edges E, and a set of parameters(probabilities) θ:

G=(N,r,E,θ)  (17)

An edge between two nodes n and n′ is denoted by (n→n′), and theprobability of moving from n to n′ along the edge is denoted by P(n′|n).A path through G begins at node r and follows exactly one outgoing edgefrom each visited node until reaching a node that contains no outgoingedges. The following properties hold of G:

-   -   1) G is a directed, acyclic graph (DAG)    -   2) ∀n ∈N·∃m·(m→n)∈E        n=r i.e. all nodes except for the root must have at least one        incoming edge.    -   3) ∃m, k∈N. ∀n∈N. (m→n)∈E        (n→k)∈E i.e. all paths branching from a given node rejoin        immediately at a subsequent common node. This property severely        constrains the structure of the graph and implies that all paths        have the same length, mitigating normalisation requirements on        path probability computations.

The context candidate model function computes the probability of a givenpath as follows (equated with the context candidate estimate):

P(c _(j)|context,M _(context-candidate))=ƒ_(context-candidate)(c_(j))=P(p _(j) |G)  (18)

where P(p_(j)|G) is the path likelihood, computed as the product of eachof the edges in the path:

P(p _(j) |G)=P(n ₁ |r)Π_(k=2) ^(K) P(n _(k) |n _(k−1))  (19)

where K is the number of edges in the path. It can be noted that thispreferred formulation amounts to an implicit independence assumptionbetween nodes. This is so because in this case the sequentialprobability of the candidate sequence is not modelled, rather theprobability of variations within the candidates is modelled. Hence, thefollowing property holds over the probabilities on edges:

∀n∈N·Σ _((n→m)∈ζ) P(m|n)=1  (20)

In other words, the probabilities on all outgoing edges from a givennode n must sum to one. This also implies that the following holds:Σ_(i)P(p_(i)|G)=1 i.e. the sum over the probability of all paths in thePCSG equals one.

An example will help to clarify these concepts. Consider the followingtwelve context candidate sequences:

“Sunday at 3pm” “sunday at 3pm” “Sun at 3pm” “Sunday at 3 pm” “sunday at3 pm” “Sun at 3 pm” “Sunday at 3p.m.” “sunday at 3p.m.” “Sun at 3p.m.”“Sunday at 3 p.m.” “sunday at 3 p.m.” “Sun at 3 p.m.”

These would be represented by the following PCSG (explicit word boundarydenoted by T and the empty sequence by ‘Ø’):

Probabilities are assigned to the edges according to the contextcandidate model, following (19), for instance:

Candidate probabilities for the twelve sequences above are thengenerated from the PCSG as follows (showing just three examples forbrevity):

-   -   P(“sunday at 3 pm”|“sunday at 3        pm”,C)=0.6*1.0*1.0*0.6*1.0*0.7=0.252    -   P(“Sunday at 3 pm”|“sunday at 3        pm”,C)=0.3*1.0*1.0*0.4*1.0*0.7=0.084    -   P(“Sun at 3 p.m.”|“sunday at 3        pm”,C)=0.1*1.0*1.0*0.4*1.0*0.3=0.012

The specifics of the model used to construct the DAG and assignprobabilities to nodes will vary depending on the particular instance ofthe system. The above schema encodes instances of three generalvariations:

-   -   branch (potentially unambiguously) at word boundaries    -   branch at case variation    -   branch at lexical variation

It will be understood that any type of variation can be encoded in thisframework. Another example would be to branch on previous suggestions,e.g. if the system had predicted both “on” and “in” and the user hadselected “in”, this can be encoded as a branch with the weight ofprobability assigned to “in” but with a small probability assigned to“on” to represent the possibility that the user accidentally acceptedthe wrong suggestion. In the above case, the following principles areencoded:

-   -   ‘sunday’ with a lower case initial ‘s’ is less probable than the        abbreviated form ‘Sun’, which is itself less probable than the        full variant ‘Sunday’.    -   The tokenisation case where “pm” is separated from the number        “3” is slightly less probable than the case where it's not.    -   The period variant “p.m.” is somewhat less probable than the        non-period form “pm”.

A particular instance of the context candidate PCSG is preferablyconstructed algorithmically from an initial sequence s in the followingmanner:

-   -   1) Turn s into a PCSG by encapsulating it within a single node        n^(s), connected to the root.    -   2) Iteratively deconstruct n^(s) by introducing branching nodes        at variation points.

For example, the PCSG construction algorithm operating on an originalsequence “sunday at 3 pm” is considered. Firstly step 1:

The system deploys a probabilistic tokeniser, resulting in thefollowing:

Note that a modification will always take the form of abranch-and-rejoin structural insertion, due to PCSG property 3 above,with the special case being a branch of one node, which is a conveniencefor subsequent processing as it does not affect overall pathprobabilities. Edge probabilities are added according to the model,which is explained in further detail below. Continuing the algorithm, acase variant analyser is deployed:

And finally a lexical variant analyser is deployed:

Note that due to PCSG property 3, branches must converge beforere-branching. This means that in some oases an empty node must beinserted, if two branch points occur contiguously.

Edge probabilities are preferably assigned to the PCSGs. The assignmentof edge probabilities is preferably carried out with respect to theparameters of the context candidate model. The intuitive interpretationof these probabilities is twofold:

-   -   1) They represent an estimate of the likelihood that the user        intended the sequence assigned to a particular branch. For        example, if a user has entered “Dear ben”, we might want to        allow, with some probability, that they actually intended to        enter “Dear Ben”.    -   2) They represent a “backoff” likelihood that a particular        branch is a valid orthographic variant of the observed sequence.        For example, if the user has entered “See you on Thur”, an        alternative orthographic form of “Thur” would be “Thurs”.

The probability assigned to a particular edge can also be influenced bythe estimated likelihood of that orthographic variant given somebackground model information. For instance, the context sequence model Scan actually be reused to obtain an estimate of the likelihood ofdifferent orthographic variants, which can be used in combination withother probabilistic measures to yield branch probabilities. Utilisingthe context sequence model in this manner means that the contextcandidate model C actually contains an instance of the context sequencemodel S, which results in an obvious violation of the independenceassumption between the candidate and sequence models (property 7 above);however, this assumption is never actually called upon in the contextcase so it is relatively safe.

An example will help to clarify. In a preferred embodiment, it isassumed that the context candidate model assigns probabilities using thefollowing algorithm:

-   -   1) Observed sequence receives 0.8 probability; others receive        the balance uniformly.    -   2) Values are scaled by the context sequence model estimates.    -   3) Values are normalised to comply with PCSG property (19)        above.

From the above PCSG example, the following branch can be considered:

As “sunday” is the original observation, it will initially be assigned aprobability of 0.8 by stage one of the above algorithm, and the otheredges will each be assigned 0.1. For the example where the estimatesreturned by the context sequence model are as follow:

-   -   P(“sunday”|C^(S))=0.01    -   P(“Sunday”|C^(S))=0.04    -   P(“Sun”|C^(S))=0.02        where C^(S) denotes that the context candidate model is        utilising the context sequence model in this case. Thus, in this        example the unnormalised and normalised (rounded) probabilities        assigned to each edge are as follows (respectively):

Context Prior Estimate

The context prior, P(context|M_(context)) can be approximated bynormalising the frequency of the original sequence t associated withcontext

$\begin{matrix}{{P\left( {context} \middle| M_{context} \right)}\overset{\sim}{=}\frac{{freq}(t)}{\sum_{t^{\prime}}{{freq}\left( t^{\prime} \right)}}} & (21)\end{matrix}$

where freq(t) is the frequency of sequence t in the training data andthe denominator is the sum of the frequencies of all sequences in thetraining data. The sequence “t” in expression (21) is the currentcontext, as input into the system. The context prior weights theprobability values of predictions according to the probability that thecorresponding model from which the prediction was drawn, comprises agiven context sequence. To do this, the context prior weights thepredictions values according to the estimate of expression (21).

In practice, this estimate would be smoothed, for example by positing anoccurrence assumption on unseen sequences, or by backing off torestricted (lower order) estimates in instances where the full sequenceis unseen. For example, if the context is a trigram, the predictionengine may be configured to back off to constituent bigram or unigramestimates.

The context prior provides a dual function: It helps to normalise theprobability estimate; and provides simple ‘model detection’ when thecontext model is unable to provide useful information. If the contextsequence estimate is uninformative (such as when the last term isunknown to an N-gram model), the context prior estimate will weight moreheavily the model with the most likely context, promoting thepredictions of this model above those from other models. The “mostlikely context” is the maximum of estimate (21) over the set of models,for example over the set of language models LM_(English), LM_(French),LM_(German) etc. For example, if the context is “The dog chased”, itwould be expected that this is significantly more likely to appear inEnglish than in French. Thus, the conditional probability of equation(21) will be greatest for LM_(English), and the probability generatorwill therefore weight the probability values of predictions drawn fromLM_(English) more heavily than those drawn from LM_(French); thusLM_(English) is ‘favoured’ by the context prior estimate.

Thus, the context prior estimate weights more heavily the mostappropriate language model from a plurality of language models relatingto a plurality of languages, given the context. In this sense, thecontext prior estimate is able to detect the language in which someoneis inputting text.

Target Sequence Prior Estimate

The target sequence prior, P(s|M_(context)) can be estimated usingsmoothed frequency analysis on the training data, in a similar way tothe context prior estimate, equation (21), e.g. the target sequenceprior can be approximated by normalising the frequency of the targetsequence over all sequences in the context training data

${P\left( s \middle| M_{context} \right)}\overset{\sim}{=}\frac{{freq}(s)}{\sum\limits_{s^{\prime}}{{freq}\left( s^{\prime} \right)}}$

where freq(s) is the frequency of the target sequence in the trainingdata and the denominator is the sum of all the frequencies of all targetsequences in the training data. The denominator can equivalently beapproximately determined as the total number of terms in the trainingdata (counting duplicates).

Input Likelihood

The input likelihood, P(input|s, M_(input)), is estimated by means ofthe first candidate model:

P(input|s,M _(input))=Σ_(j=1) ^(K) P(input|c _(j) ,M_(input-candidate))P(c _(j) |s,M _(input-sequence))  (22)

Thus, to determine the input likelihood, the following estimates need tobe computed: an input candidate estimate, P (input|c_(j),M_(input-candidate)); and an input sequence estimate. P(c_(j)|s,M_(input-sequence)). Each estimate is described below.

Input Candidate Estimate

The input candidate estimate, P(input|c_(j), M_(input-candidate)) isdefined as a function on observed input events and sequences:ƒ_(input-candidate)(i, t)=P(i|t, θ_(input-candidate)) whereθ_(input-candidate) are the parameters of the model. An arbitrary inputobservation i is encoded in an input sequence intention structure(ISIS), which is an ordered list of sets of sequences mapped toprobabilities:

{(t ₁₁ →P(i ₁ |t ₁₁),(t ₁₂ →P(i ₁ |t ₁₂) . . . }, {(t ₂₁ →P(i ₂ |t₂₁),(t ₂₂ →P(i ₂ |t ₂₂), . . . }, . . .

Note that each individual estimate has the form P(i_(j)|t_(jk)), i.e. ifthe user had intended to enter the sequence t_(jk), what's theprobability that we would have observed input event i_(j). Consider thefollowing ISIS example:

$\quad\begin{bmatrix}\left\{ {\left( H\rightarrow 0.5 \right),\left( h\rightarrow 0.3 \right),\left( g\rightarrow 0.1 \right),\left( j\rightarrow 0.1 \right)} \right\} \\\left\{ {\left( e\rightarrow 0.8 \right),\left( w\rightarrow 0.1 \right),\left( r\rightarrow 0.1 \right)} \right\}\end{bmatrix}$

This encodes the scenario in which the system estimates that had theuser intended to enter, e.g., the character ‘H’ followed by thecharacter ‘e’, so the observed input events would be expected to havethe respective probabilities of 0.5 and 0.8.

The method by which these probability distributions are generated is notthe subject of this disclosure. Rather it is highlighted that a range oftechniques are applicable, for instance:

-   -   Distributions can be generated on the basis of the characters        that surround a given target key on a particular keyboard        layout, e.g. for a QWERTY keyboard, if the user tapped the area        corresponding to the “H” key, the characters “G” and “J” might        be included in the ISIS, with some probability.    -   Distributions can be generated on the basis of the distances (or        some function of the distances, such as the square etc.) between        the touch coordinate (on a touch-screen virtual keyboard) and        the coordinates of designated keys.

In the preferred system, an input candidate is a sequence, and the setof input candidates is represented as an extended PCSG (EPCSG). An EPCSGis a PCSG but with one additional structure that violates standard PCSGproperties (defined below). As in the context case, each individualcandidate sequence is represented by a unique path through the EPCSG andthe input candidate model function return value for a given candidate iscomputed as the normalised probability of its representative path.

The input candidate EPCSG generation process starts with an ordered listof sets of sequence-probability pairs, generated by the system from userinteraction, where each subset represents a probability distributionover user input sequence intentions.

The algorithm to generate an input candidate EPCSG from an input ISIShas two stages:

-   -   1) Convert the ISIS into a PCSG.    -   2) Insert additional generalising structures, resulting in an        EPCSG.        Stage one is straightforward. Beginning with the root node of a        new PCSG, the algorithm constructs branches for each        distribution in the ISIS. The result of stage one for the above        ISIS is as follows:

Stage two embellishes the existing PCSG with two additional structures.The first is an empty node sub-path (which falls within the PCSGframework), and the second is a ‘wildcard’ structure (which converts thePCSG into an EPCSG). The following is an example application of stagetwo:

The wildcard symbol (denoted by ‘*’) is actually shorthand for a branchthat includes/generates every symbol in the character set. The wildcardstructure is a constrained cycle and therefore violates the acyclicproperty of the standard PCSG. The EPCSG extension licenses the use of awildcard cycle at convergence points only. The values e and w arepre-specified probability constants. Note that in this case each branchpoint has the empty node addition (two in this case), and eachconvergence point has the wildcard addition (one in this case). Thesegeneralisations allow for instances where the user omitted one or morecharacters from the target sequence (with the wildcard probability w) orinserted one or more erroneous characters (with the empty nodeprobability e). It will be understood that the specifics of how theseextra structures are added to the PCSG will vary with differentinstantiations of the system, depending on computational resources,sequence model strength, etc.

The empty node sub-path enables the system to disregard a characterwhich has been erroneously entered by the user, and which wouldotherwise lead to an incorrect chain through the PCGS.

With the addition of the generalisation structures (especially thewildcard branch), the number of paths through the PCSG grows rapidly.For instance, given a character set size of 50 there are 1020 uniquepaths through the simple PCSG above. For a real-world ISIS, there can betens or even hundreds of thousands of unique paths. The preferred systempreferably utilises the following techniques, alone or in anycombination, to handle this combinatorial explosion:

-   -   A trie (as known in the art) is used to ignore paths that are        not prefixes of sequences within the prediction vocabulary.    -   Probabilistic thresholding is used to prune paths that are        relatively improbable. A threshold is set on the ratio between        the current most probable sequence and the differential for less        probable sequences. Given threshold t, and a currently        investigated path length L, a path n₁ . . . n_(L) is pruned if        the following holds:

$\begin{matrix}{\frac{{P\left( n_{1} \middle| r \right)}{\prod\limits_{j = 2}^{L}{P\left( n_{j} \middle| n_{j - 1} \right)}}}{\arg \; {\max_{m}\left\lbrack {{P\left( m_{1} \middle| r \right)}{\prod\limits_{j = 2}^{L}{P\left( m_{j} \middle| m_{j - 1} \right)}}} \right\rbrack}} < t} & (23)\end{matrix}$

-   -   The input sequence model T is also used for probabilistic        thresholding. Given a distinct or tied threshold t, and a set of        sequences formed by all paths of length L: {c₁, . . . , c_(K)} a        given path p representing a particular sequence c_(p) is pruned        if the following holds:

$\begin{matrix}{\frac{P\left( c_{p} \middle| T \right)}{\arg \; {\max_{j}\left\lbrack {P\left( c_{j} \middle| T \right)} \right\rbrack}} < T} & (24)\end{matrix}$

Other techniques which are suitable for handling the combinatorialexplosion can also be employed by themselves, or in combination with oneor all of the above techniques.

Input Sequence Estimate

The input sequence estimate, P (c_(j)|s, M_(input-sequence)) is adistribution over candidate sequences given a target sequence, and canbe estimated as a normalised indicator function:

$\begin{matrix}{{P\left( {\left. c_{j} \middle| x \right.,M_{{input} - {sequence}}} \right)} = \frac{\delta \left( {s,c_{j}} \right)}{z}} & (25)\end{matrix}$

where δ(t, t′)=1 if t′ is a prefix of t and 0 otherwise, and Z=Σ_(k)δ(s,c_(k)), i.e. the sum over all candidates.

Note that if uniqueness of candidates is assumed, and the set ofcandidates is allowed to include all possible sequences, thenormalisation factor can be re-cast: Z=length(s). For example, given thetarget sequence “the”, there will always be precisely three matchingcandidates: “t”, “th” and “the”.

Thus, the present invention provides a general text prediction engineand system, and a specific example of that text prediction engine orsystem, which is configured to generate a set of sequence predictionsS_(F), each with an associated probability value P_(F).

The present invention also provides a corresponding method forprocessing user text input.

Turning to FIG. 1 and the above described system, the method comprisesreceiving text input into a user interface, e.g. of an electronicdevice; generating, using a text prediction engine, sequence predictionsS_(F) and associated probability values P_(F); and providing thesequence predictions to the user interface.

As discussed in relation to the system, the general method comprisesgenerating sequence predictions and associated probability values by atext prediction engine which comprises one or more models. In apreferred embodiment, the method comprises generating sequencepredictions from a target prior model R and from at least one modelM_(I), M₂, etc which uses at least one evidence source e₁, e₂, etc. togenerate predictions. As discussed above, in relation to the system and,in particular, expressions (12) to (15), the method comprises generatingnormalised probability values by estimating a normalisation factor forthe probability values by summing the probability values for the n mostprobable sequence predictions and a constant representative of theremaining possible sequence predictions.

With reference to FIG. 2, in the preferred embodiment, the final set ofpredictions S_(F) and associated probability values P_(F) are generatedby a probability generator PG, from sets of predictions S_(R),S_(context) S_(input) which have been drawn from a target prior model R,a context model M_(context), and an input model M_(input), respectively.In this embodiment, the context of a user entered sequence is used asevidence to draw predictions from the context model M_(context), and auser entered sequence relating to the current word the user isattempting to enter is used as evidence to draw predictions from theinput model M_(input).

Other aspects of the method are analogous to the system described above,e.g. in one embodiment of the method, the sequence predictions are onlyprovided to the user interface if their corresponding probability valuesare each greater than or equal a first threshold value.

As discussed above in relation to a system implementing generalisationstructures (and especially a wildcard branch) in a PCSG to determine acontext candidate estimate, in a preferred embodiment of the method, atleast one of the set of sequence predictions corresponds to an adjustedor corrected version of text input by a user into the user interface.

Other aspects of the method of the present invention can be readilydetermined by analogy to the above system description.

The following is a non exhaustive list of embodiments which may beclaimed in this application:

1. A system comprising:

-   -   a text prediction engine comprising:    -   at least one model configured to generate from an evidence        source a set of sequences with associated probability estimates;    -   a model configured to generate a set of sequences with        associated probability estimates; and    -   a probability generator configured to receive each set of        sequences with associated probability estimates and to generate        a set of sequence predictions with associated probability        values.        2. The system of embodiment 1 comprising a plurality of models        configured to generate from a plurality of evidence sources a        plurality of sets of sequences with associated probability        estimates.        3. The system of embodiment 1 or 2, wherein the probability        generator is configured to generate the set of sequence        predictions based on any number of independent evidence sources.        4. The system of embodiment 3, wherein one of the evidence        sources comprises user inputted text. The user inputted text can        be input by user selection, character entry, voice recognition,        etc.        5. A system comprising:    -   a user interface configured to receive text input by a user;    -   a text prediction engine according to the first aspect, or any        other suitable one, configured to receive the text input from        the user interface and to generate sequence predictions with        associated probability values;    -   wherein the text prediction engine is further configured to        provide the sequence predictions to the user interface.        6. The system of embodiment 5, wherein the text prediction        engine comprises:    -   a context model configured to receive text input by a user and        to generate a set of sequences and associated probability        estimates;    -   an input model configured to receive text input by the user and        to generate a set of sequences and associated probability        estimates;    -   a model configured to generate a set of sequences and associated        probability estimates; and    -   a probability generator configured to receive each set of        sequences and associated probability estimates from the        plurality of models and to generate the set of sequence        predictions and associated probability values.        7. The system of embodiment 6, wherein the user interface is        configured to display the sequence predictions for user        selection.        8. The system of embodiment 7, wherein the system orders the        sequence predictions by the probability values and displays them        as an ordered set.        9. The system of one of embodiment 6-8, wherein the system        preferably uses the sequence predictions to automatically        correct erroneously entered text which has been input into the        user interface.        10. The system of embodiment 6-8, wherein the text prediction        engine is configured to generate sequence predictions which are        based on any number of independent evidence sources.        11. A method for processing user text input comprising:    -   receiving text input into a user interface;    -   generating, using a text prediction engine, sequence predictions        and associated probability values;    -   providing the sequence predictions to the user interface.        12. The method of embodiment 11, wherein the method comprises        displaying the sequence predictions on the user interface for        user selection.        13. The method of embodiment 12, wherein the method comprises        ordering the sequence predictions by their associated        probability values.        14. The method of embodiment 14, comprising displaying the        ordered sequence predictions for user selection.        15. The method of any one of embodiment 11-14, comprising the        step of using the sequence predictions to automatically correct        erroneously entered text which has been input into the user        interface.        16. A text prediction engine for use in any one of the preceding        embodiments, wherein the text prediction engine comprises a        context model, an input model, a target prior model, and a        probability generator.        17. The text prediction engine of embodiment 16, wherein the        context model and input model are configured to receive the text        input by a user and to generate a set of sequences and        associated probability estimates.        18. The text prediction engine of embodiment 16 or 17, wherein        the target prior model is configured to generate a set of        sequences and associated probability estimates.        19. The text prediction engine of any of embodiments 16-18,        wherein the probability generator is configured to receive each        set of sequences and associated probability estimates from the        models and to generate a set of sequence predictions each        associated with a probability value.

It will be appreciated that this description is by way of example only;alterations and modifications may be made to the described embodimentwithout departing from the scope of the invention as defined in theclaims.

1-36. (canceled)
 37. A text prediction engine comprising: at least onemodel configured to generate from an evidence source a first set ofsequences with associated probability estimates; a probability generatorconfigured to receive the set of first sequences with associatedprobability estimates and to generate a set of sequence predictions withassociated probability values, wherein the probability values areapproximately normalised over all possible sequence predictionsgenerated by the probability generator given all of the possiblesequences received by the probability generator.
 38. The text predictionengine according to claim 37, further comprising a prior modelconfigured to generate a second set of sequences with associatedprobability estimates.
 39. The text prediction engine according to claim38, wherein: the at least one model comprises a context model and aninput model, each of which is configured to receive text input by a userand to generate a set of sequences and associated probability estimates;and the prior model comprises a target prior model configured togenerate a set of sequences and associated probability estimates. 40.The text prediction engine according to claim 39, wherein: the inputmodel comprises a candidate model and a language model; the contextmodel comprises a candidate model and a prefix match model; and thetarget prior model comprises a character model and a unigram model. 41.The text prediction engine according to claim 37, wherein the at leastone model is configured to generate a first set of sequences based onthe evidence source and an uncertainty in the evidence source.
 42. Thetext prediction engine according to claim 37, wherein the probabilitygenerator is configured to estimate a normalisation factor for theprobability values by summing the probability values for the n mostprobable sequence predictions and a constant representative of theprobability values for the remaining possible sequence predictions. 43.The text prediction engine according to claim 42, further comprising aprior model configured to generate a second set of sequences withassociated probability estimates, wherein the constant represents theprobability values for the remaining possible sequence predictionsgenerated by the at least one model and by the prior model.
 44. The textprediction engine according to claim 37, wherein the at least one modelcomprises a plurality of models configured to generate from a pluralityof evidence sources a plurality of first sets of sequences withassociated probability estimates.
 45. The text prediction engineaccording to claim 44, wherein each evidence source is modelled by acorresponding model to generate the set of sequences with associatedprobability estimates.
 46. The text prediction engine according to claim45, wherein the prediction engine is configured to weight theprobability values of the sequence predictions according to theprobability that the corresponding model comprises a given contextsequence.
 47. The text prediction engine according to claim 46, whereinone of the plurality of evidence sources comprises user inputted text,and wherein the plurality of models comprise a plurality of languagemodels corresponding to a plurality of different languages and theprediction engine is configured to weight highest the probability valuesof the sequence predictions corresponding to the language model relatingto the most likely language of user inputted text.
 48. The textprediction engine according to claim 44 wherein the probabilitygenerator is configured to treat each evidence source as conditionallyindependent of all other evidence sources, given the set of sequencepredictions.
 49. A system comprising: a user interface configured toreceive text input by a user; a text prediction engine configured toreceive the text input from the user interface and to generate a set ofsequence predictions with associated probability values, wherein theprobability values are approximately normalised over all possiblesequence predictions; wherein the text prediction engine is furtherconfigured to provide the set of sequence predictions to the userinterface.
 50. The system according to claim 49, wherein the predictionengine is configured to input automatically a sequence prediction intothe system if it has a corresponding probability value equal to orgreater than a threshold probability value.
 51. A method for processinguser text input comprising: receiving text input into a user interface;generating, using a text prediction engine, a set of sequencepredictions and associated probability values, wherein the probabilityvalues are approximately normalised over all possible sequencepredictions; providing the set of sequence predictions to the userinterface.
 52. The method according to claim 51, wherein generatingnormalised probability values comprises estimating a normalisationfactor for the probability values by summing the probability values forthe n most probable sequence predictions and a constant representativeof the remaining possible sequence predictions.
 53. The method accordingto claim 51, wherein at least one of the set of sequence predictionscorresponds to an adjusted or corrected version of text input by a userinto the user interface.
 54. The method according to claim 51, whereinthe sequence predictions are provided to the user interface only iftheir corresponding probability values are each greater than or equal afirst threshold value.
 55. The method according claim 54, furthercomprising inputting automatically a sequence prediction which has aprobability value greater than or above a second threshold value. 56.The method according to claim 51, wherein the probability generatorcomprises a plurality of models, each configured to generate a set ofsequence predictions and associated probability values, and theprobability values are weighted according to the probability that thecorresponding model comprises a given context sequence.